Maintenance support system, maintenance support method, program, method for generating processed image, and processed image

ABSTRACT

A maintenance support system prevents leakage of confidential information by excluding a maintenance target while imposing few restrictions on imaging conditions. A maintenance support system  100  includes a wearable terminal  1  including an imaging device  12 , a first identifying device  31  configured to identify a first three-dimensional area TA including a maintenance target T with reference to a predetermined reference point M, and a second identifying device  32  configured to identify a mask pixel area excluding an effective pixel area corresponding to the first three-dimensional area for a captured image acquired by the imaging device in a post-movement state in which the wearable terminal has moved. A processed-image generating device  33  is configured to generate a processed image in which the mask pixel area is made invisible and a communication device  34  is configured to transmit the processed image to a support terminal  2.

TECHNICAL FIELD

The present invention relates to a maintenance support system, amaintenance support method, a program, a method for generating aprocessed image, and a processed image for supporting maintenance workof industrial equipment which is a maintenance target. In particular,the present invention relates to a maintenance support system, amaintenance support method, a program, a method for generating aprocessed image, and a processed image, which can prevent leakage ofconfidential information etc. excluding industrial equipment which isthe maintenance target, and in which there are few restrictions onimaging conditions.

BACKGROUND ART

Conventionally, when a trouble occurs in industrial equipment such as asubstrate processing apparatus, generally action taken in responsethereto is such as follows: a maintenance worker who is at theinstallation site of the industrial equipment which is the maintenancetarget confirms the status of the trouble, and then reports the statusby telephone to a maintenance supporter of the manufacturer of themaintenance target so that the maintenance supporter gives variousinstructions to the maintenance worker by telephone.

If the trouble cannot be solved even after the above-described action bytelephone is taken, a further action is taken such that a skilled workerof the manufacturer is dispatched to the installation site of themaintenance target at a timing when the skilled worker is available.

In the case of the above-described action by telephone, since thecommunication is verbal communication, there is a case in which thestatus of the trouble may not be well communicated to the maintenancesupporter, or the instruction of the maintenance supporter cannot bewell communicated to the maintenance worker resulting in that thetrouble cannot be solved. Although, the maintenance worker may send anobtained captured image of the maintenance target to the maintenancesupporter by e-mail, etc., as needed, and the maintenance supporterchecks the captured image to grasp the status of the trouble, a problemexists in that understanding the status takes time.

Further, in the case of the above-described action in which a skilledworker is dispatched, a problem exists in that taking immediate actionis difficult.

In order to solve the problems as described above, there is proposed amaintenance support system in which a maintenance worker wears awearable terminal (head-mounted display) equipped with an imaging deviceand a display, and an image of a maintenance target is captured by usingthe wearable terminal so that this captured image (video image) istransmitted to a support terminal through a telecommunication line suchas the Internet.

However, in the above-described maintenance support system, if acaptured image is transmitted as it is to the support terminal, there isa risk that the captured image includes confidential information of theinstallation site. When the captured image including the confidentialinformation is transmitted to the maintenance supporter side, themaintenance supporter can know the confidential information, thuscausing a problem.

Accordingly, as a maintenance support system capable of preventingleakage of confidential information, for example, a maintenance supportsystem according to Patent Literature 1 (an image processing system inPatent Literature 1) has been proposed.

The system described in Patent Literature 1 is an image processingsystem (claim 1 of Patent Literature 1) including: an acquisition unitthat acquires an original image that captures an original area; arecognition unit that recognizes one or more identifiers in the originalimage; an identifying unit that identifies at least one of a first imageportion which captures a first area of the original area and a secondimage portion that captures a second area which is an area of theoriginal area excluding the first area, based on the one or moreidentifiers recognized at the recognition unit; and a generation unitthat generates a processed image including the first image portionaccording to an identified result by the identifying unit.

According to the system described in Patent Literature 1, it isconsidered that leakage of confidential information etc., can beprevented by arranging such that the first area is the maintenancetarget, and the second area is confidential information etc. excludingthe maintenance target.

However, the system described in Patent Literature 1 has a restrictionthat the original image must always include an identifier. Specifically,while Patent Literature 1 shows an example in which an internal area (arectangular area surrounded by four identifiers 30) of a printingapparatus 300 is the maintenance target, and the original area variesaccording to the movement of a worker 10 who wears a wearable tool 100including a camera 170 so that original images 171a to 173a that capturedifferent original areas are acquired, it is stated that the identifier30 must be included in each of the original images 171a to 173a(paragraphs 0063 to 0065, FIG. 6, and FIG. 7, etc., of Patent Literature1).

Therefore, for example, when an attempt is made to capture an image ofdetails of the maintenance target in close proximity to the maintenancetarget, a restriction occurs in that the maintenance target can beapproached only within a range in which the identifier is included inthe original image.

Further, since the system according to Patent Literature 1 is configuredsuch that the first image portion (the portion corresponding to themaintenance target) is two-dimensionally identified with reference tothe identifier, there is a restriction that the image of the maintenancetarget to which the identifier is bonded must be captured from aspecific one direction opposed to the identifier.

CITATION LIST Patent Literature [Patent Literature 1] JP2017-211766ASUMMARY OF INVENTION Technical Problem

An object of the present invention, which has been made to solve theabove-described problems of conventional arts, is to provide amaintenance support system, a maintenance support method, a program, amethod for generating a processed image, and a processed image, whichcan prevent leakage of confidential information etc. excludingindustrial equipment which is the maintenance target, and in which thereare few restrictions on imaging conditions.

Solution to Problem

To solve the problems, the present invention provides a maintenancesupport system comprising: a wearable terminal including an imagingdevice and worn by a maintenance worker; a first identifying deviceconfigured to identify a predetermined first three-dimensional areaincluding industrial equipment that is a maintenance target and/or toidentify a predetermined second three-dimensional area excluding theindustrial equipment that is the maintenance target, with reference to apredetermined reference point in a captured image acquired by theimaging device in an initial state; a second identifying deviceconfigured to identify a mask pixel area excluding an effective pixelarea corresponding to the first three-dimensional area and/or toidentify a mask pixel area corresponding to the second three-dimensionalarea, in the captured image acquired by the imaging device, in apost-movement state in which the wearable terminal has moved; aprocessed-image generating device configured to generate a processedimage in which the mask pixel area identified by the second identifyingdevice is made invisible for the captured image acquired by the imagingdevice in the post-movement state; and a communication device configuredto transmit the processed image generated by the processed-imagegenerating device to a support terminal operated by a maintenancesupporter.

According to the present invention, the predetermined firstthree-dimensional area including the industrial equipment which is themaintenance target and/or the predetermined second three-dimensionalarea excluding the industrial equipment which is the maintenance targetare/is identified by the first identifying device, with reference to apredetermined reference point in the captured image acquired by theimaging device in the initial state. The “imaging device in the initialstate” in the present invention means the imaging device in a state inwhich the predetermined reference point is located in the capturedimage.

Specifically, for example, when an arbitrary point of a captured imageis designated by using an AR cursor and the designated position is usedas a reference point, the “imaging device in the initial state” meansthe imaging device which is located at a position where image of thepoint designated by using the AR cursor can be captured.

Further, for example, when a pixel area corresponding to acharacteristic portion (for example, a corner portion of the maintenancetarget) in the captured image is extracted by the first identifyingdevice performing predetermined image processing on the captured image,and a center, etc., of this pixel area is used as the reference point,the “imaging device in the initial state” means the imaging devicelocated at a position where image of a characteristic portion can becaptured.

Further, for example, when the reference point is defined by a markerattached to a maintenance target or the like, the “imaging device in theinitial state” means the imaging device located at a position where animage of the marker can be captured.

Further, when the first three-dimensional area and the secondthree-dimensional area are, for example, three-dimensional areas havinga rectangular parallelepiped shape, these will be defined by thecoordinates of eight respective vertices of a three-dimensional arearepresented by the three-dimensional coordinate system with reference toa reference point.

In the above-described case, for example, by causing the coordinates ofeach vertex with reference to a reference point to be stored in advancein the first identifying device, or designating each vertex by using theAR cursor, and utilizing SLAM (Simultaneous Localization and Mapping)technique, which is known as a technique for constructing an environmentmap of the self-position and the surroundings of a sensor in an unknownenvironment, the first identifying device can identify the firstthree-dimensional area and/or the second three-dimensional area.

FIG. 10 is an explanatory diagram illustrating an outline of the SLAMtechnique.

As shown in FIG. 10, in the SLAM technique, the following first tofourth steps are carried out.

(1) First step: First, corresponding identical points (characteristicpoints) are detected in two captured images obtained by capturing imagesof the imaging target at different times t0 and t1. In the example shownin FIG. 10, corner portions of the imaging target are detected as thecharacteristic point.

(2) Second step: Next, by comparing the positions of the characteristicpoints in the two captured images, the amount of change in the postureof the imaging device (translational/rotational components) iscalculated. Further, the positions of the characteristic points in thethree-dimensional space are obtained from the combination of thecharacteristic points by the principle of triangulation. Thiscorresponds to the construction of the environment map.

(3) Third step: Further, in the two captured images obtained bycapturing the image of the imaging target at different times t1 and t2,the change in the position of the characteristic point in athree-dimensional space is obtained, and the self-position of theimaging device is estimated from this change.

(4) Fourth step: The characteristic points are registered and updated onthe three-dimensional space (updating of the environment map).

Hereinafter, in the SLAM technique, the third and fourth steps will berepeatedly carried out.

As described above, the SLAM technique is a technique for alwaysgrasping the three-dimensional space by alternately repeating theconstruction of the environment map (restoration of thethree-dimensional shape) and the estimation of the self-position of theimaging device.

The reference point in the present invention is arbitrarily set on thethree-dimensional space after the environment map is constructed (thethree-dimensional shape is restored, in other words, thethree-dimensional space is grasped).

More specific contents of the SLAM technique are described in, forexample, Information Processing Society of Japan, Technical Report Vol.2014-CVIM-190 No. 40, “3D shape restoration/self-position estimation(SLAM) and high-density 3D shape restoration from moving camera image(translation from Japanese)” and is publicly known, and thereforedetailed description thereof will be omitted in the present description.

Next, according to the present invention, in a post-movement state inwhich the wearable terminal has moved, a mask pixel area excluding theeffective pixel area corresponding to the first three-dimensional areain the captured image acquired by the imaging device is identifiedand/or a mask pixel area corresponding to the second three-dimensionalarea in the captured image acquired by the imaging device is identifiedby the second identifying device. When only the first three-dimensionalarea is identified by the first identifying device, a mask pixel areaexcluding the effective pixel area corresponding to the firstthree-dimensional area is identified by the second identifying device.Moreover, when only the second three-dimensional area is identified bythe first identifying device, a mask pixel area corresponding to thesecond three-dimensional area is identified by the second identifyingdevice. Further, when the first three-dimensional area and the secondthree-dimensional area are identified by the first identifying device, amask pixel area excluding an effective pixel area corresponding to thefirst three-dimensional area and a mask pixel area corresponding to thesecond three-dimensional area are identified by the second identifyingdevice.

As described above, since the first three-dimensional area and/or thesecond three-dimensional area with reference to the reference point inthe captured image acquired by the imaging device in the initial statehave/has been identified by the first identifying device, in thepost-movement state in which the wearable terminal has moved, the firstthree-dimensional area and/or the second three-dimensional area withreference to the reference point in the captured image acquired by theimaging device can be identified by using, for example, the SLAMtechnique. Then, upon identifying the first three-dimensional areaand/or the second three-dimensional areas in the post-movement state, areference point does not necessarily need to be included in the capturedimage acquired by the imaging device in the post-movement state.

Next, according to the present invention, a processed image in which amask pixel area identified by the second identifying device (the maskpixel area excluding the effective pixel area corresponding to the firstthree-dimensional area and/or the mask pixel area corresponding to thesecond three-dimensional area) are/is made invisible is generated by theprocessed-image generating device. Examples of the “processed image inwhich the mask pixel area is made invisible” in the present inventioninclude an image in which although the reflected area is the same as inthe captured image, the mask pixel area in the captured image is darkfilled, and an image obtained by cutting out only pixel areas other thanthe mask pixel area from a captured image.

When the mask pixel area identified by the second identifying device isa mask pixel area excluding the effective pixel area corresponding tothe first three-dimensional area, since the first three-dimensional areaincludes the maintenance target and the effective pixel area correspondsto the first three-dimensional area, the processed image in which themask pixel area excluding the effective pixel area is made invisiblewill not include a pixel area corresponding to confidential information,etc. excluding the maintenance target unless the first three-dimensionalarea is made excessively larger than the maintenance target.

Further, when the mask pixel area identified by the second identifyingdevice is a mask pixel area corresponding to the secondthree-dimensional area, the second three-dimensional area does notinclude a maintenance target. Therefore, if it is set such that thesecond three-dimensional area includes at least confidentialinformation, etc. excluding the maintenance target, a pixel areacorresponding to the confidential information, etc. excluding themaintenance target will not be included in a processed image in whichthe mask pixel area is made invisible.

Finally, according to the present invention, the processed imagegenerated by the processed-image generating device is transmitted to thesupport terminal operated by the maintenance supporter through thecommunication device. Therefore, on the support terminal, themaintenance target included in the first three-dimensional area isvisually recognized and/or the confidential information, etc. excludingthe maintenance target included in the second three-dimensional areabecomes not visually recognizable, thus making it possible to preventleakage of confidential information, etc. excluding the maintenancetarget.

As so far described, according to the present invention, it issufficient that the captured image acquired by the imaging device in theinitial state includes a reference point, and there is no need that thereference point is included in the captured image acquired by theimaging device in the post-movement state in which the wearable terminalhas moved. Further, according to the present invention, in order toidentify the predetermined first three-dimensional area including theindustrial equipment that is the maintenance target and/or to identifythe predetermined second three-dimensional area excluding the industrialequipment that is the maintenance target, there is no restriction on theimaging direction of the imaging device. Therefore, it is possible toprevent leakage of confidential information, etc. excluding themaintenance target, and there are few restrictions on the imagingconditions.

The present inventors refer to a technique of generating a processedimage in which a mask pixel area excluding the effective pixel areacorresponding to the predetermined first three-dimensional areaincluding the maintenance target and/or in which a mask pixel areacorresponding to the predetermined second three-dimensional areaexcluding the maintenance target are/is made invisible as in the presentinvention, thereby preventing leakage of confidential information etc.excluding the maintenance target included in the mask pixel area as a“Virtual Partition” technique.

The “industrial equipment” in the present invention includes those whichare defined as industrial machines by the Japan Society of IndustrialMachinery Manufacturers such as mining machinery, chemical machinery,environmental apparatuses, power transmission apparatuses, tanks,commercial washing machines, boilers/motors, plastic machinery, wind andhydraulic machinery, transport machinery, steelmaking machinery, etc.From another perspective, the “industrial equipment” in the presentinvention includes production apparatuses for producing a tangibleobject. A substrate processing apparatus is a type of industrialequipment and is also a type of production apparatus for producing atangible object in view of producing a processed substrate (for example,a substrate subjected to an etching process or a substrate subjected toa film forming process). An IT system in which electronic devices areelectrically connected via a server is not included in the productionapparatus that produces a tangible object, and is not included in theindustrial equipment either.

Further, in the present invention, the term “a post-movement state inwhich a wearable terminal has moved” means not only a state in which thewearable terminal has arrived at its moving end point (for example, aposition of the wearable terminal at the end of maintenance work), butalso a state in which it has moved to each position at eachpredetermined short time pitch in the moving process until arriving atthe moving end point. Therefore, in the processed-image generatingdevice, a processed image in which the mask pixel area is made invisiblewill be generated for all the captured images acquired in real time (ateach predetermined short time pitch) by the imaging device untilarriving at the movement end point.

In the present invention, it is also possible to adopt a mode in whichmaintenance support is performed by moving the wearable terminal suchthat not only the reference point is included in a captured imageacquired by the imaging device in the initial state, but also thereference point is always included in a captured image acquired by theimaging device in the post-movement state. However, in such a mode, samerestrictions on the imaging conditions as in the system described inPatent Document 1 will occur.

According to the present invention, it is possible to employ a mode inwhich the reference point does not need to be included in a capturedimage acquired by the imaging device in the post-movement state. Inother words, maintenance support can be performed by removing therestriction on the movement position of the wearable terminal such thatthe imaging device moves from a position where the reference pointexists to a position where the reference point does not exist within thefield of view of the imaging device.

In the present invention, for example, the reference point is defined bya marker.

When the reference point is defined by a marker, for example, the firstidentifying device is configured to grasp a three-dimensional space byusing a captured image acquired by the imaging device (for example, bythe SLAM technique by use of a captured image) and thereafter toidentify a predetermined first three-dimensional area includingindustrial equipment that is the maintenance target and/or to identify apredetermined second three-dimensional area excluding the industrialequipment that is the maintenance target, with reference to thereference point in the captured image acquired by the imaging device inthe initial state.

Preferably, the second identifying device comprises a plurality ofsecond imaging devices with imaging directions different from eachother, and identifies the mask pixel area based on captured imagesacquired by the plurality of the second imaging devices.

According to the above-described preferable configuration, since thesecond identifying device includes the plurality of second imagingdevices with imaging directions different from each other, for example,it is possible to apply the SLAM technique by use of a so-calledstereoscopic vision (stereo vision) based on the captured image acquiredby the plurality of the second imaging devices. Further, for example, byapplying the monocular SLAM technique for each captured image acquiredby the second imaging devices and combining these, it is possible toaccurately identify the mask pixel area.

Preferably, the wearable terminal is a head-mounted display including adisplay, and the head-mounted display is configured to be switchablebetween a state in which the captured image is displayed on the display,and a state in which the processed image is displayed on the display ina visually recognizable manner.

According to the above preferred configuration, in the state ofdisplaying a captured image on the display, for example, unlike the casein which the processed image in which the mask pixel area is dark filledis displayed, there is no portion that obstructs the field of view ofthe maintenance worker, and therefore it is possible to safely performthe maintenance work. The “state of displaying a captured image” in theabove-described preferable configuration means a concept including inaddition to a state of actually displaying a captured image acquired bythe imaging device on the display as in a MR head set (a state in whichthe maintenance worker visually recognizes the displayed capturedimages), a state in which the display is of a transmission type, like atransmission type AR headset, and the maintenance worker can directlysee the maintenance target through the transmission type display inwhich nothing is displayed.

On the other hand, according to the above-described preferableconfiguration, in a state of displaying a processed image on the displayin a recognizable manner, the maintenance worker can also recognize theprocessed image transmitted to the support terminal. Therefore, themaintenance worker can confirm that leakage of confidential information,etc. excluding the maintenance target is prevented, which gives a senseof relief to the maintenance worker. Note that a “state of displaying aprocessed image in a recognizable manner” in the above describedpreferable configuration includes: a state of displaying the same imageas the processed image transmitted to the support terminal (an image inwhich although the reflected area is the same as in the captured image,the mask pixel area in the captured image is dark filled, and an imageobtained by cutting out only pixel areas other than the mask pixel areafrom a captured image); and in addition to that, a state of displayingan image in which a captured image is overlaid with a boundary linebetween the mask pixel area and pixel area other than the mask pixelarea, or a state of displaying a processed image in a reduced size in apart (for example, one corner) of the display screen, and the like.

Preferably, the first identifying device, the second identifying device,the processed-image generating device, and the communication device areattached to the wearable terminal.

According to the above-described preferable configuration, all thedevices for carrying out a series of operations from generating aprocessed image in which the mask pixel area in the captured imageobtained by capturing an image of the maintenance target with theimaging device is made invisible, until transmitting this processedimage to the support terminal, is attached to the wearable terminal.This is a configuration of inside-out scheme in which the position ofthe wearable terminal (imaging device) is detected by a sensor or thelike attached to the wearable terminal itself. In the case of aconfiguration of outside-in scheme in which the position of the wearableterminal is detected by using an external device separate from thewearable terminal (for example, a configuration for detecting theposition of a wearable terminal (the position of the imaging deviceincluded in the wearable terminal) by using an imaging device or asensor installed outside the imaging device included in the wearableterminal), although there is an advantage that the power consumption ofthe wearable terminal is reduced, there are drawbacks such asenvironmental restrictions such as nonexistence of a shielding objectbetween the external device and the wearable terminal, and high costcaused by the need of a high precision external device. Therefore, byadopting the above-described preferable configuration, it is possible toachieve advantages of lower cost and fewer environmental restrictions.

Preferably, the maintenance support system according to the presentinvention further comprises the support terminal, wherein the supportterminal is configured to be capable of bidirectional data communicationwith the communication device.

According to the above-described preferable configuration, since thesupport terminal and the communication device are capable of conductingbidirectional data communication, not only it is possible to transmitprocessed image from the communication device to the support terminal,but also it is possible to perform effective maintenance work by solelyusing the maintenance support system according to the present invention,such as transmitting image for maintenance work from the supportterminal to the communication device, and transmitting voice dataindicating the contents of the maintenance work.

In the above-described preferable configuration, more preferably, themaintenance support system further comprises a server which iselectrically connected to the communication device and the supportterminal, and in which first identification information and secondidentification information are stored, the first identificationinformation being identification information of the maintenance workerand the second identification information being identificationinformation of the maintenance supporter, wherein the server determineswhether or not identification information transmitted from thecommunication device to the server matches the first identificationinformation stored in the server, and also determines whether or notidentification information transmitted from the support terminal to theserver matches the second identification information stored in theserver, and enables bidirectional data communication between thecommunication device and the support terminal via the server when theidentification information transmitted from the communication device tothe server matches the first identification information, and theidentification information transmitted from the support terminal to theserver matches the second identification information.

According to the above-described more preferable configuration, theserver determines whether or not the identification informationtransmitted from the communication device to the server matches thefirst identification information stored in the server (identificationinformation of the maintenance worker). In other words, the serverdetermines whether or not the identification information of themaintenance worker who uses the maintenance support system according tothe present invention has been transmitted to the server. Further, theserver determines whether or not the identification informationtransmitted from the support terminal to the server matches the secondidentification information (identification information of themaintenance supporter) stored in the server. In other words, the serverdetermines whether or not the identification information of themaintenance supporter who uses the maintenance support system accordingto the present invention has been transmitted to the server. Then, whenthe identification information transmitted from the communication deviceto the server matches the first identification information, and theidentification information transmitted from the support terminal to theserver matches the second identification information (in other words, itis authenticated that both the identification information of themaintenance worker who uses the maintenance support system according tothe present invention, and the identification information of themaintenance supporter have been transmitted to the sever),bi-directional data communication between the communication device andthe support terminal via the server is enabled.

According to the above-described more preferable configuration, onlywhen it is authenticated that both the identification information of themaintenance worker who uses the maintenance support system according tothe present invention, and the identification information of themaintenance supporter have been transmitted to the sever, bi-directionaldata communication between the communication device and the supportterminal is enabled so that only the maintenance workers and maintenancesupporters who are determined in advance can use the maintenance supportsystem according to the present invention, and therefore it is possibleto more surely prevent leakage of the confidential information, etc.excluding industrial equipment which is the maintenance target.

Note that in addition to the above-described preferable configuration orthe above-described more preferable configuration, if the configurationis made such that data communication is enabled from industrialequipment (for example, substrate processing apparatuses), which is themaintenance target, to the support terminal, and if for example, processlog data obtained at the maintenance target (measurement values andsetting values concerning the process of the maintenance target) istransmitted to the support terminal, the maintenance supporter caninstruct contents of maintenance work while referring to these processlog data, and perform further effective maintenance work.

To solve the above-described problems, the present invention alsoprovides a maintenance support method by use of a wearable terminalincluding an imaging device and worn by a maintenance worker, themaintenance support method comprising: a first identifying step ofidentifying a predetermined first three-dimensional area includingindustrial equipment which is a maintenance target and/or identifying apredetermined second three-dimensional area excluding the industrialequipment which is the maintenance target, by a first identifyingdevice, with reference to a predetermined reference point in a capturedimage acquired by the imaging device in an initial state; a secondidentifying step of identifying a mask pixel area excluding an effectivepixel area corresponding to the first three-dimensional area and/oridentifying a mask pixel area corresponding to the secondthree-dimensional area, in the captured image acquired by the imagingdevice, by a second identifying device, in a post-movement state inwhich the wearable terminal has moved; a processed-image generating stepof generating, by a processed-image generating device, a processedimage, wherein the mask pixel area identified in the second identifyingstep is made invisible for the captured image acquired by the imagingdevice in the post-movement state; and a communication step oftransmitting, by a communication device, the processed image generatedin the processed-image generating step to a support terminal.

To solve the above-described problems, the present invention alsoprovides a program for causing the first identifying device, the secondidentifying device, the processed-image generating device and thecommunication device to respectively execute the first identifying step,the second identifying step, the processed-image generating step and thecommunication step included in the above-described maintenance supportmethod.

Note that the present invention can also be provided as a computer (CPU)readable storage medium which stores the above-described program.

To solve the above-described problems, the present invention alsoprovides a method for generating a processed image, comprising: in acaptured image acquired by an imaging device included in a wearableterminal worn by a maintenance worker, identifying a mask pixel areaexcluding an effective pixel area corresponding to a predetermined firstthree-dimensional area including industrial equipment which is amaintenance target and/or identifying a mask pixel area corresponding toa predetermined second three-dimensional area excluding the industrialequipment which is the maintenance target, and generating a processedimage in which the mask pixel area is made invisible.

To solve the above-described problems, the present invention alsoprovides a method for generating a processed image by use of a capturedimage acquired by an imaging device included in a wearable terminal wornby a maintenance worker, the method comprising: a first identifying stepof identifying a predetermined first three-dimensional area includingindustrial equipment which is a maintenance target and/or identifying apredetermined second three-dimensional area excluding the industrialequipment which is the maintenance target, by a first identifyingdevice, with reference to a predetermined reference point in a capturedimage acquired by the imaging device in the initial state; a secondidentifying step of identifying a mask pixel area excluding an effectivepixel area corresponding to the first three-dimensional area and/oridentifying a mask pixel area corresponding to the secondthree-dimensional area, in a captured image acquired by the imagingdevice, by a second identifying device, in a post-movement state inwhich the wearable terminal has moved; and a processed-image generatingstep of generating, by processed-image generating device, a processedimage in which the mask pixel area identified in the second identifyingstep is made invisible for a captured image acquired by the imagingdevice in the post-movement state.

To solve the above-described problems, the present invention alsoprovides a processed image wherein, in a captured image acquired by animaging device included in a wearable terminal worn by a maintenanceworker, a mask pixel area excluding an effective pixel areacorresponding to a predetermined first three-dimensional area includingindustrial equipment which is a maintenance target and/or a mask pixelarea corresponding to a predetermined second three-dimensional areaexcluding the industrial equipment which is the maintenance targetare/is made invisible.

To solve the above-described problems, the present invention furtherprovides a processed image generated by the above-described generatingmethod, wherein a mask pixel area excluding an effective pixel areacorresponding to the first three-dimensional area including industrialequipment which is the maintenance target and/or wherein a mask pixelarea corresponding to the second three-dimensional area excluding theindustrial equipment which is the maintenance target are/is madeinvisible.

Advantageous Effects of Invention

According to the present invention, it is possible to achieve effects ofpreventing leakage of confidential information, etc. excluding themaintenance target, and reducing restrictions on imaging conditions.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 (FIGS. 1A to 1C) is a diagram to schematically show an outlineconfiguration of a maintenance support system according to a firstembodiment of the present invention.

FIG. 2 is an explanatory diagram to illustrate outline operation of themaintenance support system shown in FIG. 1.

FIG. 3 (FIGS. 3A and 3B) is a diagram to show an example of a capturedimage acquired by the imaging device in the initial state shown in FIG.1 and an example of a processed image generated by the processed-imagegenerating device.

FIG. 4 (FIGS. 4A and 4B) is a diagram to show an example of a capturedimage acquired by the imaging device in the post-movement state in whichthe wearable terminal shown in FIG. 1 has moved, and an example of aprocessed image generated by the processed-image generating device.

FIG. 5 is an explanatory diagram to illustrate a setting example of afirst three-dimensional area in a maintenance support system accordingto a second embodiment of the present invention.

FIG. 6 is an explanatory diagram to illustrate a secondthree-dimensional area to be used in a maintenance support systemaccording to a third embodiment of the present invention.

FIG. 7 (FIGS. 7A and 7B) is an explanatory diagram to illustrate asetting example of a first three-dimensional area and outline operationof the second identifying device at this time in a maintenance supportsystem according to a fourth embodiment of the present invention.

FIG. 8 (FIGS. 8A to 8C) is a plan view to illustrate outline operationof the second identifying device when the first three-dimensional areais set as shown in FIG. 7.

FIG. 9 (FIGS. 9A and 9B) is a diagram to show an example of a processedimage which is generated by the processed-image generating device when afirst three-dimensional area is set, and also a second three-dimensionalarea is set as shown in FIG. 7.

FIG. 10 is an explanatory diagram to illustrate an outline of the SLAMtechnique.

DESCRIPTION OF EMBODIMENTS

Hereinafter, maintenance support systems according to embodiments of thepresent invention will be described with reference to the appendeddrawings.

First Embodiment

FIG. 1 is a diagram to schematically show an outline configuration of amaintenance support system according to a first embodiment of thepresent invention. FIG. 1A is a general configuration diagram, FIG. 1Bis a diagram showing a wearable terminal and components attachedthereto, and FIG. 1C is a block diagram showing the internal structureof the control-signal processing device shown in FIG. 1B.

As shown in FIG. 1A, a maintenance support system 100 according to thepresent embodiment includes a wearable terminal 1, a support terminal 2,and a control-signal processing device 3. Further, the maintenancesupport system 100 includes a server 6 as a preferable configuration.

The wearable terminal 1 is a terminal worn by a maintenance worker whoperforms maintenance work at the installation site of a maintenancetarget T (industrial equipment such as a substrate processingapparatus).

As shown in FIG. 1B, the wearable terminal 1 of the present embodimentis a glasses-type wearable terminal (head-mounted display), and includesa frame 11 that a maintenance worker puts on his/her ears. The wearableterminal 1 also includes an imaging device 12 at the middle of the frontside of the frame 11 (a position corresponding to the vicinity of themiddle of forehead of a maintenance worker when the maintenance workerputs the frame 11 on his/her ears). A color (RGB) camera is used as theimaging device 12 of the present embodiment. An imaging direction(line-of-sight direction) L1 of the imaging device 12 is set to be adirection directly facing the middle of forehead of the maintenanceworker when the maintenance worker puts the frame 11 on his/her ears.Further, the wearable terminal 1 includes a display 13 which is attachedto the front side of the frame 11 and covers a front field of view ofthe maintenance worker when the maintenance worker puts the frame 11 onhis/her ears. As the display 13 of the present embodiment, one of atransmission type is used. Therefore, when nothing is displayed on thedisplay 13, it can be used in the same manner as with ordinary glasses.

Note that the imaging device 12 can be used as a hardware portion of thesecond identifying device 32 to be described later, as needed.

As shown in FIG. 1B, a plurality of second imaging devices 32 a and 32 bwith imaging directions different from each other are attached to thewearable terminal 1 of the present embodiment. Specifically, in thepresent embodiment, a pair of second imaging devices 32 a and 32 b areattached to the left and right end portions on the front side of theframe 11, and an imaging direction (line-of-sight direction) L2 of thesecond imaging device 32 a and an imaging direction (line-of-sightdirection) L3 of the second imaging device 32 b are set to directionsdifferent from each other. As the second imaging devices 32 a and 32 b,for example, a color (RGB) camera can be used as in the imaging device12. Further, as the second imaging devices 32 a and 32 b, for example, awide-angle camera including a wide-angle lens (including anultra-wide-angle lens and a fisheye lens) can be used. Note that thesecond imaging devices 32 a and 32 b can be used as a hardware portionof the second identifying device 32 to be described later.

As shown in FIG. 1B, a range image sensor (also referred to as a depthcamera or a 3D camera) 32 c having the substantially same line-of-sightdirection as the imaging device 12 is attached to the wearable terminal1 of the present embodiment. Specifically, the range image sensor 32 cis attached below the imaging device 12 at the middle of the front sideof the frame 11. As the range image sensor 32 c, a range image sensorthat measures a distance to an object in the field of view by, forexample, the TOF (Time Of Flight) method and acquires a range image inwhich the distance to the object is represented by a gray level of eachpixel. Note that the range image sensor 32 c can be used as a hardwareportion of the second identifying device 32 to be described later.

Further, an inertial sensor 32 d is attached to the wearable terminal 1of the present embodiment. Specifically, the inertial sensor 32 d isattached to a side of the frame 11. The inertial sensor 32 d includes,for example, a triaxial acceleration sensor, a triaxial angular velocitysensor (gyro sensor), and a triaxial geomagnetic (azimuth) sensor. Notethat the inertial sensor 32 d can be used as a hardware portion of thesecond identifying device 32 to be described later.

Further, a speaker 4 and a microphone 5 are attached to a side of theframe 11 of the wearable terminal 1 of the present embodiment.

The wearable terminal 1 having the above-described configuration can beconstructed by using, for example, “HoloLens” (registered trademark),which is smart glasses manufactured by Microsoft Corporation, andimproving the same.

The support terminal 2 is a terminal which is generally located at aposition remote from the installation site of the maintenance target Tand is operated by the maintenance supporter. The support terminal 2 iselectrically connected to the wearable terminal 1 (specifically, thecontrol-signal processing device 3 attached to the wearable terminal 1)through a telecommunication line N such as the Internet. Specifically,the support terminal 2 of the present embodiment is electricallyconnected to the wearable terminal 1 via a server 6 electricallyconnected to the telecommunication line N. Further, the support terminal2 is configured to be capable of bidirectional data communication withthe communication device 34 to be described later. As the supportterminal 2, for example, a desktop computer can be used, but withoutbeing limited to this, various terminals such as a laptop computer, atablet computer, a smartphone, etc., can be used as far as they candisplay processed images as described below.

Note that the support terminal 2 of the present embodiment iselectrically connected to the maintenance target T as well through thetelecommunication line N, but this is not an essential configuration.Specifically, in the present embodiment, for example, a computer (notshown in FIG. 1A) that collects process log data of the maintenancetarget T (measurement values and setting values related to the processof the maintenance target T) is provided, and this computer iselectrically connected to the support terminal 2 via the server 6electrically connected to the telecommunication line N. Thus, aconfiguration is made such that the process log data obtained at themaintenance target T is sequentially transmitted to the support terminal2 and is stored in the support terminal 2. By providing the server 6, itbecomes easy to manage the identification information and the usagehistory, of the wearable terminal 1 and the maintenance target T whichare connected to the support terminal 2.

The control-signal processing device 3, which is attached to thewearable terminal 1 (on a side of the frame 11), is electricallyconnected to each component of the imaging device 12, the display 13,the second imaging devices 32 a and 32 b, the range image sensor 32 c,the inertial sensor 32 d, the speaker 4 and the microphone 5, and hasfunctions of controlling each component and processing an output signalof each component.

The control-signal processing device 3 is mainly composed of a CPU, amemory such as ROM and RAM, and a program stored in the memory andcausing the CPU to execute the operation to be described later.

As shown in FIG. 1C, the control-signal processing device 3 includes aprogram for causing the CPU to execute the operation as the firstidentifying device 31, a program for causing the CPU to execute theoperation of the software portion 32 e of the second identifying device32, and a program for causing the CPU to execute the operation as theprocessed-image generating device 33. Further, the control-signalprocessing device 3 includes a communication device 34. Thecommunication device 34 is composed of an antenna, a program foroperating the antenna, and the like.

Updating of the program as described above may be performed by directlyconnecting a predetermined computer and the control-signal processingdevice 3 with a USB cable or the like, or may be performed by updatingdirectly from the server 6.

In the present embodiment, all of the first identifying device 31, thesecond identifying device 32 (the software portion 32 e and at least oneof the imaging device 12, the second imaging devices 32 a and 32 b, therange image sensor 32 c, and the inertial sensor 32 d), theprocessed-image generating device 33, and the communication device 34are attached to the wearable terminal 1. As described above, this is aconfiguration of inside-out scheme in which the position of the wearableterminal 1 (the imaging device 12) is detected by a sensor or the likeattached to the wearable terminal 1 itself.

Hereinafter, operation (maintenance support method) of the maintenancesupport system 100 according to the present embodiment having theoutline configuration described above, will be described with referenceto FIGS. 2 to 4, in addition to FIG. 1.

First, when using the maintenance support system 100 according to thepresent embodiment, the server 6 authenticates both the identificationinformation of the maintenance worker and the identification informationof the maintenance supporter. Specifically, the server 6 determineswhether or not both the identification information of the maintenanceworker and the identification information of the maintenance supporterare transmitted to the server 6.

First identification information which is the identification informationof the maintenance worker and second identification information which isthe identification information of the maintenance supporter are storedin the server 6 in advance before the maintenance support system 100 isused. Specifically, for example, the maintenance supporter inputs thefirst identification information (for example, ID number) of themaintenance worker who is determined to use the maintenance supportsystem 100 into the support terminal 2 by using the keyboard of thesupport terminal 2 or the like so that the input first identificationinformation is transmitted to the server 6 via the telecommunicationline N and is stored. Similarly, for example, the maintenance supporterinputs the second identification information (for example, ID number) ofthe maintenance supporter who is determined to use the maintenancesupport system 100 to the support terminal 2 by using the keyboard ofthe support terminal 2 or the like so that the input secondidentification information is transmitted to the server 6 through thetelecommunication line N and is stored. Alternatively, a configurationmay be such that the first identification information of the maintenanceworker who may use the maintenance support system 100 and the secondidentification information of the maintenance supporter who may use themaintenance support system 100 are stored in advance in the supportterminal 2 so that the maintenance supporter selects the firstidentification information of the maintenance worker who actually usesthe maintenance support system 100 and the second identificationinformation of the maintenance supporter who actually uses themaintenance support system 100, out of the first identificationinformation and the second identification information stored in thesupport terminal 2 by using a keyboard or the like of the supportterminal 2, and thereby the selected first identification informationand the second identification information are transmitted to the server6 through the telecommunication line N and are stored therein.

Note that in the present embodiment, when the first identificationinformation and the second identification information are stored by theserver 6, use-start times and usage times of the maintenance supportsystem 100, which are used by the maintenance worker and the maintenancesupporter, are also stored together in the server 6.

When the use-start time of the maintenance support system 100 comes orthe use-start time approaches, identification information is transmittedfrom the communication device 34 of the wearable terminal 1 to theserver 6. The identification information transmitted from thecommunication device 34 is, for example, the identification informationof the wearable terminal 1 stored in advance in the control-signalprocessing device 3 of the wearable terminal 1. In this case, thewearable terminal 1 worn by the maintenance worker is determined inadvance, and the identification information of the wearable terminal 1is transmitted to the server 6 as the identification information (firstidentification information) of the maintenance worker. Similarly, whenthe use-start time of the maintenance support system 100 comes or theuse-start time approaches, the identification information is transmittedfrom the support terminal 2 to the server 6. This identificationinformation transmitted from the support terminal 2 is, for example, thesecond identification information of the maintenance supporter input bythe maintenance supporter using the keyboard or the like of the supportterminal 2. When the maintenance supporter who operates the supportterminal 2 is determined in advance, the identification information ofthe support terminal 2 stored in advance in the support terminal 2 maybe transmitted to the server 6 as the identification information of themaintenance supporter (the second identification information).

The server 6 determines whether or not the identification informationtransmitted from the communication device 34 to the server 6 asdescribed above matches the first identification information stored inthe server 6. Similarly, the server 6 determines whether or not theidentification information transmitted from the support terminal 2 tothe server 6 as described above matches the second identificationinformation stored in the server 6. Then, when the identificationinformation transmitted from the communication device 34 to the server 6matches the first identification information, and the identificationinformation transmitted from the support terminal 2 to the server 6matches the second identification information (when it is authenticatedthat both the identification information of the maintenance worker whouses the maintenance support system 100 and the identificationinformation of the maintenance supporter who uses the maintenancesupport system 100 have been transmitted to the server 6), thebidirectional data communication between the communication device 34 andthe support terminal 2 via the server 6 is enabled until the usage timeelapses from the use-start time.

As described above, in the maintenance support system 100 according tothe present embodiment, only when it is authenticated that both theidentification information of the maintenance worker and theidentification information of the maintenance supporter have beentransmitted to the server 6, bidirectional data communication betweenthe communication device 34 and the support terminal 2 is enabled.Therefore, only the maintenance worker and the maintenance supporter,who are determined in advance, are allowed to use the maintenancesupport system 100, and therefore it is possible to prevent leakage ofthe confidential information, etc. excluding the maintenance target Tmore reliably.

Note that the configuration may be such that bidirectionaltransmission/reception of voice data is enabled at the time whenbidirectional data communication between the communication device 34 andthe support terminal 2 is enabled as described above, and thetransmitted/received voice data is stored in the server 6 and/or thesupport terminal 2. By storing the voice data, it is possible toreliably save the history of maintenance support.

Further, in the case where at least one of the maintenance worker andthe maintenance supporter stops or interrupts the use of the maintenancesupport system 100 before the usage time elapses, the configuration maybe made such that predetermined instruction information that indicatesstopping or interruption of the use is transmitted from thecommunication device 34 of the wearable terminal 1 and/or the supportterminal 2 to the server 6 so that the server 6 stops or interrupts thebidirectional data communication between the communication device 34 andthe support terminal 2.

Furthermore, when there are a plurality of wearable terminals 1 or whenthere are a plurality of support terminals 2, it may be configured toadopt a mode in which all the bidirectional data communication in eachterminal 1 and 2 is enabled only when all the identification informationtransmitted from each terminal 1 and 2 is authenticated by the server 6,or to adopt a mode in which only the bidirectional data communication ineach authenticated terminal 1 and 2 is enabled.

FIG. 2 is an explanatory diagram to illustrate outline operation of themaintenance support system 100 according to the present embodiment. InFIG. 2, the maintenance worker wearing the wearable terminal 1 is notshown. FIG. 3 is a diagram to show an example of a captured imageacquired by the imaging device 12 in the initial state and an example ofprocessed image generated by the processed-image generating device 33.FIG. 3A shows an example of the captured image, and FIG. 3B shows anexample of the processed image. FIG. 4 is a diagram to show an exampleof a captured image acquired by the imaging device 12 in thepost-movement state in which the wearable terminal 1 has moved, and anexample of a processed image generated by the processed-image generatingdevice 33. FIG. 4A shows an example of the captured image, and FIG. 4Bshows an example of the processed image. Note that, in FIG. 3 and FIG.4, a boundary line of a first three-dimensional area TA is illustratedby a broken line for convenience of explanation, but the boundary linedoes not exist in an actual captured image.

As shown in FIG. 2, members T1 to T3 are provided on the front side ofthe maintenance target T of the present embodiment. Next to themaintenance target T, an apparatus S related to confidential informationis disposed.

First, the maintenance worker wearing the wearable terminal 1, forexample, moves to a position directly facing the maintenance target T soas to be able to capture an image of the maintenance target T with theimaging device 12 of the wearable terminal 1. Then, the maintenanceworker captures an image of the maintenance target T with this imagingdevice 12 in the initial state (the imaging device 12 included in thewearable terminal 1 shown by a solid line in FIG. 2) to acquire thecaptured image (video image) as shown in FIG. 3A.

Next, for example, the maintenance worker designates an arbitrary pointon a captured image displayed on the display 13 by using an AR cursor,and thereby the first identifying device 31 identifies theabove-described designated point as a reference point M and stores it inthe first identifying device 31. Further, it is also possible that thefirst identifying device 31 performs predetermined image processing on acaptured image, and thereby extracts a pixel area corresponding to acharacteristic portion (for example, a corner portion, etc., of themaintenance target T) in the captured image to identify and store acenter, etc., of this pixel area as the reference point M. Further, whena marker is attached to the maintenance target T (a marker such as an ARmarker, a QR code (registered trademark), and a bar code is attached bybonding, printing, imprinting, etc.), it is also possible that the firstidentifying device 31 performs predetermined image processing on acaptured image and thereby extracts a pixel area corresponding to themarker and identifies a center, etc., of the pixel area as the referencepoint M.

Then, in the present embodiment, the first identifying device 31identifies a predetermined first three-dimensional area (for example, arectangular parallelepiped three-dimensional area) TA including themaintenance target T with reference to the reference point M. The firstthree-dimensional area TA is a three-dimensional area in a virtualspace. The coordinates of each vertex P1 to P8 of the firstthree-dimensional area TA represented by three-dimensional coordinatesystem with reference to the reference point M (the three-dimensionalcoordinate system having X, Y, and Z axes shown in FIG. 2) are stored inadvance in the first identifying device 31, and the first identifyingdevice 31 uses, for example, the SLAM technique to identify the firstthree-dimensional area TA based on the coordinates of each of thesevertices P1 to P8.

Moreover, it is also possible that the maintenance worker designateseach vertex P1 to P8 of the first three-dimensional area TA by using anAR cursor on a captured image displayed on the display 13, and therebythe first identifying device 31 identifies the first three-dimensionalarea TA by using, for example, the SLAM technique. In this case, sinceeach vertex P1 to P8 of the first three-dimensional area TA is disposedin the three-dimensional coordinate system with reference to thereference point M in an instance that it is designated by use of the ARcursor, the coordinates of each vertex P1 to P8 are determined and as aresult, the first three-dimensional area TA can be identified.

Further, when a marker is attached to the maintenance target T, it isalso possible that the first identifying device 31 calculates acharacteristic quantity of the pixel area corresponding to the marker toidentify identification information of the marker from thecharacteristic quantity, and identifies the first three-dimensional areaTA by using, for example, the SLAM technique based on the coordinates ofeach vertex P1 to P8 stored in association with the identifiedidentification information in the first identifying device 31. When, inthis way, the reference point M is defined by a marker, sincecoordinates of each vertex P1 to P8 have been already stored in thefirst identifying device 31, the maintenance worker does not need to setthem so that not much effort will be required. When the shape of themaintenance target T is complicated, it is possible to accurately seteach vertex P1 to P8 (or, vertices of more than eight) based on thedesign drawing of the maintenance target T in advance so as to beassociated with the identification information of the marker, which isparticularly effective.

Note that FIG. 2 exemplifies a case in which the wearable terminal 1worn by the maintenance worker moves within the first three-dimensionalarea TA.

Next, as the maintenance worker wearing the wearable terminal 1 moves,the wearable terminal 1 moves as well, and in a post-movement state inwhich the wearable terminal 1 has moved, an image of the maintenancetarget T is captured with the imaging device 12 included in the wearableterminal 1 (the imaging device 12 included in the wearable terminal 1illustrated by a broken line in FIG. 2) to acquire a captured image asshown in FIG. 4A. This captured image does not include a reference pointM.

Then, in the above-described moving process, captured images arecontinuously acquired by the imaging device 12 and the second imagingdevices 32 a and 32 b. Moreover, range images are continuously acquiredby the range image sensor 32 c. Further, acceleration, angular velocity,and geomagnetism (azimuth) of the imaging device 12 included in thewearable terminal 1 are continuously detected by an inertial sensor 32d.

The second identifying device 32 identifies a mask pixel area excludingan effective pixel area corresponding to the first three-dimensionalarea TA in a captured image acquired by the imaging device 12 in apost-movement state in which the wearable terminal 1 has moved.

Specifically, when the second imaging devices 32 a and 32 b constitutethe second identifying device 32, based on the captured imagecontinuously acquired by the second imaging devices 32 a and 32 b in themoving process, a software portion 32 e constituting the secondidentifying device 32 constructs an environmental map of theself-position and the surroundings of the wearable terminal 1 (theimaging device 12), and thereby identifies the first three-dimensionalarea TA in a captured image in the post-movement state.

Moreover, when the range image sensor 32 c constitutes the secondidentifying device 32, based on the range image continuously acquired bythe range image sensor 32 c in the moving process, the software portion32 e constituting the second identifying device 32 identifies the firstthree-dimensional area TA in a captured image in the post-movement stateby using, for example, the SLAM technique.

Further, the same applies to a case in which the imaging device 12 andthe inertial sensor 32 d constitute the second identifying device 32.Moreover, there is a conceivable case in which any of the second imagingdevices 32 a and 32 b, the range image sensor 32 c, the imaging device12 and the inertial sensor 32 d are combined to constitute the secondidentifying device 32, and in such a case, the second identifying device32 identifies the first three-dimensional area TA in the captured imagein the post-movement state by using, for example, the SLAM technique.

Note that the second identifying device 32 of the present embodimentdoes not use GPS in identifying the first three-dimensional area TA in acaptured image in the post-movement state (as described later, the sameapplies to a case of identifying the second three-dimensional area SA ina captured image in the post-movement state). This is because of thatthe GPS cannot be used indoor, and that position detection error of atleast several meters occurs.

The second identifying device 32 identifies, as described above, thefirst three-dimensional area TA in a captured image in the post-movementstate, and thereby identifies the mask pixel area excluding theeffective pixel area corresponding to the first three-dimensional areaTA. It is possible by geometric calculation to identify which portionthe first three-dimensional area TA is reflected within the field ofview of the imaging device 12 in the post-movement state, in otherwords, which pixel area it corresponds to in a captured image acquiredby the imaging device 12. Accordingly, it is possible to identify aneffective pixel area corresponding to the first three-dimensional areaTA and to identify a mask pixel area excluding the effective pixel area.

Next, the processed-image generating device 33 generates a processedimage (video image) in which the mask pixel area identified by thesecond identifying device 32 is made invisible for the captured imageacquired by the imaging device 12 in the post-movement state. In anexample shown in FIG. 4B, an image in which a mask pixel area is darkfilled is generated as a processed image. However, the processed imagemay be, without being limited to this, an image in which only theeffective pixel area (only pixel area excluding the mask pixel area) iscut out from the captured image. Moreover, instead of dark filling themask pixel area, various modes may be adopted as long as leakage ofconfidential information can be prevented such as by making the areamosaicked and fogged.

Finally, the communication device 34 transmits a processed imagegenerated by the processed-image generating device 33 to the supportterminal 2 through a telecommunication line N. Therefore, as shown inFIG. 4A, even if an apparatus S related to confidential information isvisible in a captured image, on the support terminal 2, as shown in FIG.4B, only the maintenance target T included in the firstthree-dimensional area TA is visually recognized, and thus it ispossible to prevent leakage of confidential information, etc. excludingmaintenance target T.

Note that the second identifying device 32, the processed-imagegenerating device 33, and the communication device 34 operate on acaptured image acquired by the imaging device 12 in the initial state asshown in FIG. 3A. Thereby, in the processed-image generating device 33,a processed image as shown in FIG. 3B in which the mask pixel area ismade invisible will be generated even for a captured image acquired bythe imaging device in the initial state and transmitted to the supportterminal 2.

Note that the second imaging devices 32 a and 32 b or the like whichconstitute the second identifying device 32 fails or causes any trouble,it becomes not possible to correctly identify the change in the positionof the imaging device 12 in the post-movement state from the initialstate. Accordingly, it becomes not possible to correctly identify aneffective pixel area corresponding to the first three-dimensional areaTA in a captured image acquired by the imaging device 12 in thepost-movement state. Consequently, there arises possibility that agenerated processed image includes confidential information etc.excluding the maintenance target T.

Therefore, it is preferable that the processed-image generating device33 has a function of generating a processed image in which all the pixelareas are made invisible, or the communication device 34 has a functionof stopping transmission of the processed image, when the second imagingdevices 32 a and 32 b or the like fails or causes any trouble. Further,at this time, it is preferable that the fact that failure or sometrouble has occurred is displayed on the display 13, or the fact thatfailure or some trouble has occurred is notified by voice to the speaker4 so that the maintenance worker can promptly know that failure or sometrouble has occurred in the second imaging devices 32 a and 32 b or thelike.

The display 13 of the present embodiment is configured to be switchablebetween a state in which a captured image is displayed on the display 13(specifically, in the present embodiment, a state in which themaintenance worker can directly visually recognize the maintenancetarget T through the transmission-type display 13 on which nothing isdisplayed) and a state in which a processed image is displayed on thedisplay 13 in a recognizable manner. Specifically, for example, it ispossible to adopt a mode in which a switching button (not shown) isprovided on the frame 11 and the control-signal processing device 3controls to select either one of the states by pressing the switchingbutton.

In a state in which a captured image is displayed on the display 13 (astate in which the maintenance worker can directly visually recognizethe maintenance target T), since there is no portion that obstructs thefield of view of the maintenance worker, maintenance work can beperformed safely.

On the other hand, in a state in which a processed image is displayed onthe display 13 in a recognizable manner, the maintenance worker can alsorecognize the processed image transmitted to the support terminal 2.Therefore, the maintenance worker can also confirm that leakage of theconfidential information, etc. excluding the maintenance target T isprevented, which gives a sense of relief to the maintenance worker. Notethat examples of the state in which processed image is displayed in arecognizable manner include, a state in which the same thing (an imagein which although reflected area is the same as a captured image, themask pixel area in a captured image is dark filled, an image obtained bycutting out only an effective pixel area from a captured image, and thelike) as the processed image transmitted to the support terminal 2 isdisplayed, as well as a state in which an image obtained by overlaying aboundary line between the mask pixel area and the effective pixel area(boundary line of the first three-dimensional area TA) on the capturedimage is displayed as shown in FIGS. 3A and 4A, and a state in which aprocessed image is reduced in size and displayed on a portion of thedisplay screen.

Note that in the present embodiment, although description has been madeon a case as an example in which one first three-dimensional area TA isidentified for one maintenance target T, the present invention will notbe limited to this. For example, when there are a plurality ofmaintenance targets T, a reference point M may be set for eachmaintenance target T and the first three-dimensional area TA may beidentified for each maintenance target T. For example, when the onemaintenance target T is a substrate processing apparatus, and parts ofthe substrate processing apparatus are removed and repair work etc., isperformed on a predetermined work desk, another maintenance target T isset to this work desk, and the reference point M may be set on the workdesk as well.

Further, in the present embodiment, as the first three-dimensional areaTA, a three-dimensional area having a rectangular parallelepiped shapehaving eight vertices P1 to P8 has been described as an example, but thepresent invention is not limited to this. Depending on the shape of themaintenance target T, it is also possible to set the firstthree-dimensional area TA having a shape other than the rectangularparallelepiped, which is identified by more than eight points.

Further, in the present embodiment, a case in which one wearableterminal 1 is electrically connected to the support terminal 2 has beendescribed as an example, but the present invention will not be limitedto this. It is also possible to adopt a configuration in which aplurality of wearable terminals 1 are electrically connected to thesupport terminal 2.

In this case, the configuration may be such that the imaging device 12of the plurality of wearable terminals 1 identify the same firstthree-dimensional area TA with reference to the same reference point M,or it is also possible to adopt a configuration in which a differentreference point M is used, or a different first three-dimensional areaTA is identified for each wearable terminal 1.

Further, the support terminal 2 can adopt various modes such as one inwhich the processed images transmitted from the plurality of wearableterminals 1 are displayed side by side at the same time, or theprocessed image to be displayed is switchable.

Furthermore, it is possible to prepare a plurality of support terminals2 electrically connected to the wearable terminal 1, regardless ofwhether the wearable terminal 1 is singular or plural.

Further, in the present embodiment, although the area including themaintenance target T and also including the range in which the wearableterminal 1 moves is set as the first three-dimensional area TA as shownin FIG. 2, the present invention will not be limited to this. For anarea in which non-existence of an apparatus S related to confidentialinformation is determined in advance even if it is an area in which themaintenance target T does not exist or an area in which the wearableterminal 1 does not move, it is also possible to further widen the firstthree-dimensional area TA so as to include such area. Further, the firstthree-dimensional area TA is not limited to one continuous area, and aplurality of separated areas can be set as the first three-dimensionalarea TA.

Second Embodiment

A maintenance support system according to a second embodiment has asimilar outline configuration to that of the maintenance support system100 according to the first embodiment which has been described withreference to FIG. 1.

However, the maintenance support system according to the secondembodiment is different in the setting of the first three-dimensionalarea TA from the maintenance support system 100 according to the firstembodiment. Hereinafter, description will be made on this point.

FIG. 5 is an explanatory diagram to illustrate a setting example of thefirst three-dimensional area TA in the maintenance support systemaccording to the second embodiment.

As shown in FIG. 5, in the present embodiment, the firstthree-dimensional area TA consists of a plurality (eight in the exampleshown in FIG. 5) of virtual partition walls W1 to W8. In the exampleshown in FIG. 5, an area surrounded by the eight partition walls W1 toW8 is set as the first three-dimensional area TA. The partition walls W1to W8 may be designated by the maintenance worker using an AR cursor forthe captured image displayed on the display 13. As a result, the firstidentifying device 31 identifies the area surrounded by the partitionwalls W1 to W8 as the first three-dimensional area TA.

The partition walls as shown in FIG. 5 are preferably configured suchthat the position, size, and number thereof can be designated freely bythe maintenance worker. According to this preferable configuration, evenif the shape or the size of the maintenance target T changes, theposition, size, and number of the partition wall may be freely changed,and therefore, such configuration provides versatility, as well as makesit easier to identify the mask pixel area by the second identifyingdevice 32. Further, for example, when the maintenance target T includesnot only a main body of the substrate processing apparatus but also anauxiliary apparatus (for example, a pump or a chiller) located at aplace slightly apart from the main body of the substrate processingapparatus, it is possible to more appropriately arrange partition wallsto configure the first three-dimensional area TA as compared to thefirst three-dimensional area TA of a constant size as shown in FIG. 2(the layout can be easily changed).

Note that a small gap may exist between the partition walls as long asthe apparatus S related to confidential information cannot be visuallyrecognized on the support terminal 2 (that is, invisible in theprocessed image). FIG. 5 illustrates an example in which some gaps existbetween the partition walls W1 and W2, between the partition walls W2and W3, between the partition walls W5 and W6, and between the partitionwalls W6 and W7. However, in order to ensure that the apparatus Srelated to confidential information cannot be visually recognized on thesupport terminal 2, it is preferable that each partition wall has aportion overlapping with each other as seen from the imaging direction(line-of-sight direction) L1 of the imaging device 12 of the wearableterminal 1 within the first three-dimensional area TA.

Further, in the example shown in FIG. 5, no partition wall is providedin the ceiling portion or the floor portion of the place where themaintenance target T is located, but the partition walls may be setfreely as long as the apparatus S related to confidential informationaround the maintenance target T is not visible on the support terminal2.

Third Embodiment

A maintenance support system according to a third embodiment also hasthe same outline configuration as that of the maintenance support system100 according to the first embodiment described with reference to FIG.1.

However, the maintenance support system according to the thirdembodiment is different from the maintenance support system 100according to the first embodiment in that a second three-dimensionalarea SA is used instead of the first three-dimensional area TA.Hereinafter, this point will be described.

The maintenance support system according to the third embodiment isconfigured such that the first identifying device 31 identifies apredetermined second three-dimensional area excluding the maintenancetarget T with reference to the reference point M (for example,identifies the second three-dimensional area SA including the apparatusS related to confidential information as shown in FIG. 6), and thesecond identifying device 32 identifies a mask pixel area correspondingto the second three-dimensional area SA. The second three-dimensionalarea SA is a three-dimensional area in a virtual space. In the case ofidentifying the second three-dimensional area SA as well, the contentsof description on the case of identifying the first three-dimensionalarea TA described above can be similarly applied.

The first identifying device 31 may identify the secondthree-dimensional area SA instead of identifying the firstthree-dimensional area TA, or identifying both the firstthree-dimensional area TA and the second three-dimensional area SA. Whenthe first identifying device 31 identifies the second three-dimensionalarea SA instead of identifying the first three-dimensional area TA, thesecond identifying device 32 will identify a mask pixel areacorresponding to the second three-dimensional area SA. When the firstidentifying device 31 identifies both the first three-dimensional areaTA and the second three-dimensional area SA, the second identifyingdevice 32 identifies a mask pixel area excluding an effective pixel areacorresponding to the first three-dimensional area TA, and identifies amask pixel area corresponding to the second three-dimensional area SA.Note that when the both areas are identified, the mask pixel areaexcluding the effective pixel area corresponding to the firstthree-dimensional area TA and the mask pixel area corresponding to thesecond three-dimensional area SA may overlap. In this case, a pixel areawhich is identified as a mask pixel area at least in one area may beidentified as the mask pixel area.

Fourth Embodiment

A maintenance support system according to a fourth embodiment also hasthe same outline configuration as that of the maintenance support system100 according to the first embodiment described with reference to FIG.1.

However, the maintenance support system according to the fourthembodiment differs from the maintenance support system 100 according tothe first embodiment and the maintenance support system according to thesecond embodiment in the setting of the first three-dimensional area TA.Hereinafter, this point will be described. In the following description,a case in which there are two apparatuses (apparatuses S1 and S2)related to confidential information will be exemplified.

FIG. 7 is an explanatory diagram to illustrate a setting example of thefirst three-dimensional area TA and an outline operation of the secondidentifying device 32 at this time in the maintenance support systemaccording to the fourth embodiment. FIG. 7A is a perspective view toshow a setting example of the first three-dimensional area TA. FIG. 7Bis a plan view to illustrate the outline operation of the secondidentifying device 32, and to show a state in which the wearableterminal 1 is located at a position directly facing the middle of thefront side (the side on which the members T1 to T3 shown in FIG. 7A areprovided) of the maintenance target T (the imaging direction L1 of theimaging device 12 of the wearable terminal 1 shown in FIG. 7A isorthogonal to the front side of the maintenance target T).

FIG. 8 is a plan view to illustrate an outline operation of the secondidentifying device 32 when the first three-dimensional area TA is set asshown in FIG. 7. FIG. 8A shows a state in which the wearable terminal 1is in a position moved in parallel from the state shown in FIG. 7B tothe apparatus S2 side. FIG. 8B shows a state in which the wearableterminal 1 is in a position rotationally moved (moved around themaintenance target T in a counterclockwise direction) from the stateshown in FIG. 7B to the apparatus S2 side. FIG. 8C shows a state inwhich the wearable terminal 1 turned to the left on the spot from thestate shown in FIG. 7B.

Note that, in FIG. 7B and FIG. 8, the wearable terminal 1 is illustratedas a rectangle for convenience.

In the example shown in FIG. 2 described above, an area including themaintenance target T and also including a range in which the wearableterminal 1 moves is set as the first three-dimensional area TA, but inthe example shown in FIG. 7A, the first three-dimensional area TA is setso as to match the maintenance target T and the wearable terminal 1moves outside the first three-dimensional area TA.

In a state in which the wearable terminal 1 is located at a positiondirectly facing the middle of the front side of the maintenance target Tas shown in FIG. 7B, when an image of the maintenance target T (firstthree-dimensional area TA) is captured by the imaging device of thewearable terminal 1, an object located in a range of the angle of view θof the imaging device of the wearable terminal 1 forms an image and isacquired as a captured image. In other words, an object located in anoutside area of imaging OIA, which is outside of a range of the angle ofview θ of the imaging device of the wearable terminal 1 shown in FIG.7B, will not constitute a captured image.

The second identifying device 32 identifies a mask pixel area excludingan effective pixel area corresponding to the first three-dimensionalarea TA in an acquired captured image. Specifically, the secondidentifying device 32 identifies, in the captured image, a pixel areaobtained as a result of an object, which is located in the mask area MAshown in FIG. 7B, forming an image, or a pixel area obtained as a resultof an object forming an image through the mask area MA, as the maskpixel area.

An object located in a dead angle area BA, which is located behind thefirst three-dimensional area TA (maintenance target T) as seen from theimaging device of the wearable terminal 1, is blocked by the maintenancetarget T and does not form an image, and therefore will not constitute acaptured image. This results in the same effect as in the case ofidentifying a pixel area excluding an effective pixel area correspondingto the first three-dimensional area TA, as the mask pixel area.

Moreover, when an image of the first three-dimensional area TA(maintenance target T) is captured by the imaging device of the wearableterminal 1, an area which is present between the wearable terminal 1 andthe first three-dimensional area TA, and is also present within a rangeof angle of view θ of the imaging device is originally an area otherthan the first three-dimensional area TA, but the second identifyingdevice 32 identifies this area as an exception area EA located in frontof the first three-dimensional area TA. Then, the second identifyingdevice 32 identifies a pixel area, which is obtained as a result of anobject forming an image through this exception area EA, as an effectivepixel area, in a captured image. Since the pixel area obtained by imageforming through the exception area EA (that is, a pixel areacorresponding to the front side of the maintenance target T) is aneffective pixel area, the processed-image generating device 33 will notmake it invisible. In other words, in the processing, priority is givento visual recognizability of the first three-dimensional area TA(maintenance target T).

As shown in FIG. 8A, when the wearable terminal 1 is moved in parallelfrom a state shown in FIG. 7B to the apparatus S2 side (when themaintenance worker wearing the wearable terminal 1 moves to the rightside while looking at the maintenance target T from the front), theposition of the exception area EA greatly changes compared to the stateshown in FIG. 7B. In the example shown in FIG. 8A, since the apparatusS2 is located in the mask area MA, the second identifying device 32identifies the pixel area obtained by image forming of the apparatus S2in the captured image as the mask pixel area, and the processed-imagegenerating device 33 makes this mask pixel area invisible so that theapparatus S2 will not be reflected in the processed image. However,depending on the position of the wearable terminal 1, the apparatus S2may fall within the exception area EA. In such a case, it is alsopossible to separately set the above-described second three-dimensionalarea SA.

As shown in FIG. 8B, when the wearable terminal 1 moves by rotation tothe apparatus S2 side from the state shown in FIG. 7B (when themaintenance worker wearing the wearable terminal 1 moves around themaintenance target T counterclockwisely) as well, the apparatuses S2 andS1 may fall within the exception area EA as in the case of FIG. 8A. Asdescribed before, it is possible to cope with such case by separatelysetting a second three-dimensional area SA.

As shown in FIG. 8C, when the wearable terminal 1 changes itsorientation to the left on the spot from the state shown in FIG. 7B (themaintenance worker wearing the wearable terminal 1 changes its viewingdirection (orientation of its head) to the left direction on the spot),if, as shown in FIG. 7B, the apparatus S1 or S2 is originally locatedwithin the dead angle area BA, it will remain to be located within thedead angle area BA even if the line-of-sight direction is changed.

As in the maintenance support system according to the fourth embodimentdescribed with reference to FIGS. 7 and 8, when a firstthree-dimensional area TA is set so as to match the maintenance targetT, and the wearable terminal 1 moves outside the first three-dimensionalarea TA, it is necessary to separately set the second three-dimensionalarea SA in preparation for the possibility that the apparatus S (S1 andS2) related to confidential information is located in an exception areaEA: however, as in the case shown in FIG. 2 described above, on thesupport terminal 2, only the first three-dimensional area TA(maintenance target T) is visually recognized, and it is possible toprevent leakage of confidential information, etc. excluding themaintenance target T.

FIG. 9 is a diagram to show an example of the processed image which isgenerated by the processed-image generating device 33 when the firstthree-dimensional area TA is set as shown in FIG. 7, as well as thesecond three-dimensional area SA is set. FIG. 9A is a diagram toillustrate the imaging direction (line-of-sight direction) L1 of theimaging device 12 of the wearable terminal 1, and FIG. 9B is a diagramto show an example of the processed image which is generated at the timeof the imaging direction L1 shown in FIG. 9A. Note that in FIG. 9A,illustration of the wearable terminal 1 is omitted.

As shown in FIG. 9B, a processed image in which the mask pixel areaexcluding the effective pixel area corresponding to the predeterminedfirst three-dimensional area TA including the maintenance target T (themaintenance target T=the first three-dimensional area TA), and the maskpixel area corresponding to the second three-dimensional area SA (notshown) respectively set for the apparatuses S1 and S2 shown in FIG. 9Aare made invisible (dark filled in the case shown in FIG. 9) in thecaptured image acquired by the imaging device 12 included in thewearable terminal 1 worn by the maintenance worker.

Note that in the viewpoint of layout, when an apparatus S related toconfidential information is not present in the vicinity of themaintenance target T (that is, there is no apparatus S related toconfidential information between the area in which the worker wearingthe wearable terminal 1 moves and the maintenance target T), setting ofthe first three-dimensional area TA will be satisfactory, and setting ofthe second three-dimensional area SA will be needed when an apparatus Srelated to confidential information is present in the vicinity of themaintenance target T.

REFERENCE SIGNS LIST

-   1 Wearable terminal-   2 Support terminal-   3 Control-signal processing device-   12 Imaging device-   13 Display-   31 First identifying device-   32 Second identifying device-   32 a, 32 b Second imaging device-   32 c Range image sensor-   32 d Inertial sensor-   33 Processed-image generating device-   34 Communication device-   100 Maintenance support system-   M Reference point-   N Telecommunication line-   S, S1, S2 Apparatus related to confidential information-   T Maintenance target-   TA First three-dimensional area-   SA Second three-dimensional area-   W1, W2, W3, W4, W5, W6, W7, W8 Partition wall-   θ Angle of view-   BA Dead angle area-   OIA Outside area of imaging-   EA Exception area-   MA Mask area

1. A maintenance support system comprising: a wearable terminalincluding an imaging device and worn by a maintenance worker, a firstidentifying device configured to identify a predetermined firstthree-dimensional area including industrial equipment that is amaintenance target and/or to identify a predetermined secondthree-dimensional area excluding the industrial equipment that is themaintenance target, with reference to a predetermined reference point ina captured image acquired by the imaging device in an initial state; asecond identifying device configured to identify a mask pixel areaexcluding an effective pixel area corresponding to the firstthree-dimensional area and/or to identify a mask pixel areacorresponding to the second three-dimensional area, in the capturedimage acquired by the imaging device, in a post-movement state in whichthe wearable terminal has moved; a processed-Image generating deviceconfigured to generate a processed image in which the mask pixel areaidentified by the second identifying device is made invisible for thecaptured image acquired by the imaging device in the post-movementstate; and a communication device configured to transmit the processedimage generated by the processed-image generating device to a supportterminal operated by a maintenance supporter.
 2. The maintenance supportsystem according to claim 1, wherein there is no need that referencepoint is included in the captured image acquired by the imaging devicein the post-movement state.
 3. The maintenance support system accordingto claim 1, wherein the reference point is defined by a marker.
 4. Themaintenance support system according to claim 1, wherein the secondidentifying device comprises a plurality of second imaging devices withimaging directions different from each other, and identifies the maskpixel area based on captured images acquired by the plurality of thesecond imaging devices.
 5. The maintenance support system according toclaim 1, wherein the wearable terminal is a head-mounted displayincluding a display, and the head-mounted display is configured to beswitchable between a state in which the captured image is displayed onthe display, and a state in which the processed image is displayed onthe display in a visually recognizable manner.
 6. The maintenancesupport system according to claim 1, wherein the first identifyingdevice, the second identifying device, the processed-image generatingdevice, and the communication device are attached to the wearableterminal.
 7. The maintenance support system according to claim 1,further comprising: the support terminal, wherein the support terminalis configured to be capable of bidirectional data communication with thecommunication device.
 8. The maintenance support system according toclaim 7, further comprising: a server which is electrically connected tothe communication device and the support terminal, and in which firstidentification information and second identification information arestored, the first identification information being identificationinformation of the maintenance worker and the second identificationinformation being identification information of the maintenancesupporter, wherein the server determines whether or not identificationinformation transmitted from the communication device to the servermatches the first identification information stored in the server, andalso determines whether or not identification information transmittedfrom the support terminal to the server matches the secondidentification information stored in the server, and enablesbidirectional data communication between the communication device andthe support terminal via the server when the identification informationtransmitted from the communication device to the server matches thefirst identification Information, and the identification informationtransmitted from the support terminal to the server matches the secondidentification information.
 9. A maintenance support method by use of awearable terminal including an imaging device and worn by a maintenanceworker, the maintenance support method comprising: a first identifyingstep of identifying a predetermined first three-dimensional areaincluding industrial equipment which is a maintenance target and/oridentifying a predetermined second three-dimensional area excluding theindustrial equipment which is the maintenance target, by a firstidentifying device, with reference to a predetermined reference point ina captured image acquired by the imaging device in an initial state; asecond identifying step of identifying a mask pixel area excluding aneffective pixel area corresponding to the first three-dimensional areaand/or identifying a mask pixel area corresponding to the secondthree-dimensional area, in the captured image acquired by the imagingdevice, by a second identifying device, in a post-movement state inwhich the wearable terminal has moved; a processed-image generating stepof generating, by a processed-image generating device, a processedimage, wherein the mask pixel area identified in the second identifyingstep is made invisible for the captured image acquired by the imagingdevice in the post-movement state; and a communication step oftransmitting, by a communication device, the processed image generatedin the processed-image generating step to a support terminal.
 10. Acomputer readable storage medium which stores a program for causing thefirst identifying device, the second identifying device, theprocessed-image generating device and the communication device torespectively execute the first identifying step, the second identifyingstep, the processed-image generating step and the communication stepincluded in the maintenance support method according to claim
 9. 11. Amethod for generating a processed image, comprising: in a captured imageacquired by an imaging device included in a wearable terminal worn by amaintenance worker, identifying a mask pixel area excluding an effectivepixel area corresponding to a predetermined first three-dimensional areaincluding industrial equipment which is a maintenance target and/oridentifying a mask pixel area corresponding to a predetermined secondthree-dimensional area excluding the industrial equipment which is themaintenance target, and generating a processed image in which the maskpixel area is made invisible. 12-14. (canceled)